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ABSTRACT 


Let  <.X,Y),  (X.  ,Y,  ) . (X  ,Y  )  be  i.i.d.  R’^xR-valued  random  vectors 

i  i  n  n 

with  E|Y|  <  00,  and  let  be  a  kernel  estimate  of  the  regression  function 

Q(x)  =  E(Y|X  =  x).  In  this  paper,  we  establish  an  exponential  bound  of  the 
mean  deviation  between  and  Q(x)  given  the  training  sample  z”  = 

iX^ ,Y. , . . . ,X  ,Y  ),  under  the  conditions  as  weak  as  possible. 


1.  INTRODUCTION 


Let  (X,Y),  (X, ,Y, (X  ,Y  )  be  i.i.d.  R  xR-valued  random  vectors  with 
i  i  n  n 

E|Y!  <  «•  To  estimate  QCx)  *  E(Y|X=x),the  regression  function  of  Y  with 
respect  to  X,  Nadaraya  (1964)  and  Watson  (1964)  proposed  a  class  of  kernel 
estimates  of  the  form 

On'**  -  I 

i=l  j=l 

where  K  is  a  probability  density  ftinction  on  R  ,  and  h=h^  is  a  sequence  of 
positive  numbers.  Write 

W  (x)  =  K(^^)/  y  K(-i^),  (2) 

ni  h  L  h 

j=l 


we  define  i=1.2,...,n,  when  0/0  appears.  Many  scholars  studied 

convergence  problems  of  these  estimates  from  various  points  of  view.  For 
the  universal  consistency,  one  can  refer  to,  for  example,  Devroye  and  Wagner 
(19801),  Spiegelman  and  Sacks  (1980).  For  the  pointwise  moment-consistency, 
see  Devroye  (1981).  For  the  pointwise  a.s.  consistency,  see  Devroye  (1981), 
Greblicki,  Krzyzak  and  Pawlak  (1984),  Zhao  and  Fang  (1985).  In  this  paper, 
we  study  another  convergence  of  these  estimates. 

Let  =  (Xj^,Yj^ . X^,Y^)  be  a  training  sample,  g^  =  g^(x,z’^)  be  an 

estimate  of  Q(x).  In  some  problems,  we  are  interested  in  the  following 
mean  deviation  of  g^  given  the  training  sample  z’^; 


D(8„)  =  E{ig„(X,z")  -  Q(X)i |Z"} 


f  -|g_(x,z”)  -  Q(x)|F(dx), 


(3) 


where  F  denotes  the  distribution  of  X.  Devroye  and  Wagner  (1980II)  proved 


lira  .  ..D(Q  )  =  0  a.s. 
n-«»  n 


for  the  kernel  estimates  of  Q(x),  if  the  following  conditions  are 


satisfied: 


(.i)  Y  is  bounded, 

(ii)  F  has  a  density  f, 

(iii)  K  is  bounded  and 
J  ij)(x)dx  <  «, 


where 


i|»(x)  =  Sup 


K(u).  xeR’^ 


u  >  X 


and  I  I .  i  I  is  the  norm  or  L  norm  on 


(iv)  h  -*0  and  S  exp(-cxnh’^)  <  »  for 
any  a  >  0. 

In  this  paper,  we  establish  an  exponential  bound  for  the  above  mentioned 

deviation  of  Q  .  Take  I  I ■ I  I  as  or  L  norm,  and  denote  by  1(A)  or  I.  the 
n  '  '  '  ^  00  '  A 

indicator  of  set  A.  We  establish  the  following 

Theorem.  Let  Qjj(x)  b  a  kernel  estimate  defined  by  (1).  Suppose  that  the 
following  conditions  are  satisfied: 

(i)  Y  is  bounded. 

(ii)  F,  the  distribution  of  X,  has  a 
density  of  f. 

(iii)  There  exist  positive  constant  a 

and  0  such  that 
o 

K(x)  i  al(  I  (x|  l<^o^),  xcR’^. 

( iv )  h-K)  and  nh^-*<”  as  n-*«> . 


Then  for  any  given  e  >  0,  we  have 


P{D(Q  )  >e}  <  e’^^".  (6 

n  ““ 

where  C  >  0  is  a  constant  independent  of  n. 

Obviously,  we  need  only  to  give  the  proof  for  norm.  We  shall 

indtroduce  some  lemmas  in  section  2,  and  give  a  proof  of  the  theorem  in 
section  3. 


Accesion  For  ^ 
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Availability  Codes 


•  Avail  and/or 


2.  SOME  LEMMAS 


For  simplicity  we  use  the  following  convention:  e  .  ,C,Cq,Cj^, 

. . .  ,a,Sj^,32i'5.  ect.,  are  all  constants  independent  of  n-  //(A)  denotes  the 
cardinal  of  set  A.  S  =  {u6R  :  1 lu-x( o}.  F  and  X  denote  the  outer 

X  t  p 

measure  generated  by  F  and  the  Lebesque  measure  X(on  R  ),  respectively.  F^ 
denotes  the  empirical  measure  of  x”  =  .  ,X^) .  We  now  give  four  lemmas 

which  are  needed  in  the  sequel. 

Lemma  KBesicovitch  Covering  Lemma).  Let  E  be  a  bounded  subset  of  R^, 
and  let  K  be  a  family  of  cubes  covering  E  which  contain  a  cube  with 
center  x  for  each  xfiE.  Then  there  exist  points  {x^^}  in  E  such  that 
(  i )  E  c  u  D  , 


Proof .  Set 


For  any  xcE,  there  exists  a  cube  S  with  d^(0,1)  such  that  A(S  ) 

1  x,p  x,p 

>  F(S  )  >  F(S  )/S, .  By  Lemma  1  there  exist  x,  eE-  and  S  AS,  such 

x,p  x,p  1  k  1  *lc’^k  '  ^ 

that  >  F(Sj^)/Bj^,  Ej^  k^k  5.  Thus 

^'k 

F*(E^)  <  F(USj^)  <  I^F(Sj^)  <  bJ'US^) 

Taking  >  0  small  enough,  we  have  F(Ej^)<e/2.  In  the  same  way,  we  have 

\  (E^)  <  F(R^)a/S2  ®  c/3t  Taking  large  enlugh,  we  can  make  A  (E^)  small 

* 

enough  and,  by  the  absolute  continuity  of  F  with  respect  to  A,  F  (.E2)  <  e/2. 
The  lemma  is  proved. 


Fix  6«(0,l/2a)  and  assume  that  h  =*  h  e(0,l).  Set 

n 

<-  V^x.h'  ‘ 


(8) 


Lemma  3.  Suppose  that  F  has  a  density  f,  h  =  h^e(0,l)  and  nn  T^en 
for  any  given  e  >  0  we  have 

*  *  1 ^ 

P{F  (G^)  >  e}  <  e  . 


Proof.  By  Lemma  1,  there  exist  x,eG  and  S,  =  S  ,  such  that 

iC  I\  iC  " 


V 


G  U  S,  “  G,  )I(S,  )  ^  o.  Partition  into  sets  with  the  form 
n  K  K  *  K 

r 

jnj^[(ij-l)eh,ijeh),  where  ij^,...,i^  =  0,±1,±2,...  and  e  is.  a  fixed  constant  to 
be  chosen  later.  Call  the  partition  ♦,  and  write 

-  {Bt*:  B  c  Sg^2T^* 

^’VG,Be*®’  ^0,1*  n  t-l+e,l-e). 


c  =  S,  -  Un,.  D^c  B  C  X,  +  h(S»  ,-S»  ,) 

Xj^  k  —  K  0,1  0,1 


,\(C^  )  «  =  (2h)’^[l-(l-e)’^] 


Since 


we  see  that 


reA(Sj^) . 


A(G-G)  <  y  X(C*  )  <  rey  X(S.  ) 

^■k  *k  ^k 

<  rej^jg  yi(Sj^)dA  <  reoA(Uj^Sj^) 

<  reoA(SQ 

Hence  we  can  choose  e  small  enough  to  render  A(G-G)  small  enough  and 
F(G-G)  <  e/4.  By  (.8)  and  the  fact  that  o6  <  1/2,  we  get 

F  (G)  <  F  (G)  <  y  .F  (S,  )  <  6y  F(S,  ) 
n  •"  n  n  k  k 

=  sj^jg  yi(Sj^)dF  1  (SoF(USj^)  -  5oF(G) 


<  iF(G), 


Therefore 


F(G)  -  F  (G)  >  F(G)  -  e/4  -  iF(G)  -  iF(G)  -  c/4, 
n 


F  (G  )  >  e  implies  F(G)  -  F  (G)  >  e/4, 
n  ~  n 


For  any  He  t',  we  write  uH  ■  U,^B.  Then 


{F  (G^)>c}  c  u  f^^^-{F(UH)-F^(UH)>e/4}. 


Assume  that  e€(0,l).  By  Hoeffding's  inequality. 


Sup  P{F(A)-F  (A)>e/4}  <  Sup  exp{-n(e/4)  / [2F( A)+e/4] } 
An  A 

<  exp{-ne^/[16(2+e/4]}  <  exp( -ne^/36) . 

C  h’^ 

Noticing  that  £  2  °  and  h  ^  =  o(n),  we  get 

P{F*(G*)>e}  <  2  °  Sup.P{F(A)-F  (A)>e/4> 
n  ~  “  A  n  ~ 

^  exp(-Cj^n), 

and  the  lemma  is  proved. 

Lemma  4.  Suppose  that  f  jg(x) |^F(dx)«»  for  some  p  >  0,  then 


lim^jg  ig(u)-g(x)  t^F(du)/F(S^^j^)  =  0 

X  t  h 

for  almost  all  x(with  respect  to  F). 
Refer  to  Wheeden  and  Zygmund  (1977),  p.  191,  example  20. 


3.  PROOF  OF  THE  THEOREM 


Assume  that  |Y|  f.  M.  Without  loss  of  generality,  we  can  assume  that 
0  =  1  in  (5)(iii).  It  is  enough  to  prove  that  for  each  fixed  T  >  0, 


P{L  IQ  (x)  -  Q(x)lF(dx)  >  £}  <  e‘‘^". 
J^O.T  " 


(9) 


By  Lemma  2,  there  exist  g^  =  S2(e)  and  a  compact  set  EcE  such 


that  F(S  ^-E)  <  e/8M,  wher  E  is  defined  by  (7).  Hence 

o,T 


js  1  2MFiSQ  ^-E)  <  e/4. 

0 ,  T 

Fix  6t(0,l/2o).  By  Lemma  3,  there  exists  a  compact  set  H  such  that 


H 


(10) 


and 


-C  n 

P{F(S.  „-H„)  >  £/(8M)}  <  e  ^ 
u ,  i  n  ~ 


(11) 


Hence 


P{ 


_  „  |Q  (x)  -  Q(x)|F(dx)  >  e/4}  <  e 

^0,T'  n  " 


-Cj^n 


Therefore,  we  need  only  to  prove  that 

P{[h  oE^^n^*^  "  Q(*nP(dx)  >  e/2}  <  e 


-C2n 


(12) 


For  xeH  HE,  bv  (5)(iii),  (7)  and  (10),  we  have 
n 


X.“x 


K(-;i — )  >  naF  (S  .  )  >  na6F(S  ,  ) 
L  h  -  n'  x,h'  -  x,h'^ 


j=l 


>  0068^^2^^^^, 


and  f(x)  £  g^.  Write  =  g.,/(a6gj^2^) ,  we  see  that 


r 


IH  OE'^n^*^  '  Q(x)lf(x)«lx 
n  n 


1  C3(nh’^)'^|H  I  [  K(^VnY.-QCx/)ldx. 


There  exist  finite  positive  constants  and  disjoint  regular  cubes 

it  lU 

A. , . . . ,A  such  that  K  (x)  *  E  a.I,  (x)  satisfies 
i  m  .  .  1  A. 

1=1  1 


[  |K(x)  -  K'‘(x)|dx  <  e/(8C-M). 

Here  a  regular  cube  means  a  r-fold  product  of  one-dimensional  compact  intervals 
Thus 

,  •  S  X.-x  ^  X.-x 

P{(nh’^)"-^  H  HE  I  /  ^  (-^))(Y.-Q(x))idx 

1=1 

.  ^  r  X.-x  ^  X.-x 

<  aC^Mlnh^)"^  ^  ^ 

i=l 

<  2C3m||K(x)  -  K*(x)|dx  <  g/4. 

Take  =  z/iAC^).  To  prove  (12) ♦  it  is  enough  to  prove  that 

^  X  *x 

P{(nh^)'^jjj  I  y  K(-^)(Y.-Q(x))idx  >  e^} 

"  i=l 


It  is  sufficient  for  any  >  0  and  any  regular  cube  A  to  prove  that 

n 

r,E  I  I  i  '2> 


li 

J  r*  ■  t-» 


We  proceed  to  prove  (13).  To  this  end,  we  construct  the  pratition  $ 

r 

of  R  mentioned  in  the  proof  of  Lemma  3.  Assume  that  A  =  n  [x.,x.+a.]  and 

r  i«l  ^  ^  ^ 

min  a.  i  2e.  Set  A  -  I  {x.+e.x.+Aj-e). 


C  *x+hA-Acx+  h(A-A)  ■  C  . 

X  X  X 

It  is  easy  to  see  that  we  can  make  A(A-A)  arbitrarily  small  by  choosing  e 

small  enough.  We  have 

n 

‘"•'"'■‘Ih  he  I  I 

"  i-1 

n 


<  (nh 


|H  (IE  '  <Xi)<V^-0(X^))idx 


+  2Mh“^|l 

-  ^  jH^nE/^B€*,Bcx+hA  I  \  I3(X^)(Y^-Q(X^) )  jdx 


i»l 


F  (C  )<ix 
'  n  X 


(15) 


+  2MA(A-A). 


Here  we  use  the  fact  that  Jv(x+hD)dx  =  h’^A(D)  for  any  r-dimensional  probability 

measure  v  and  any  Borel  set  DcR^.  We  can  choose  e  such  that  2MA(A-A)<e2/2. 

Note  that  for  Be*,  A{x.Bcx+hA}  <  C^h’^,  and  ^g{x+hA}cs  ^  for  small  h. 

n  ’*■ 

Hence,  for  large  n,  we  have 

n 

(15)  <  I  Ib(X^)(Y.-Q(X^))|  +  zj2. 

i-1 

Set  £3  *  £2/(40^).  To  prove  (13),  we  need  only  to  prove  that 


C  r  "Ccn 

1  ^  ■  "JB  <  e  ^  , 

(16) 

i-1 

n  ^  ^ 

1  ^  Ig(X^)Q(X^)  -  n|g  Q(x)dFl>ne3}  <  e  '  ^  . 

(17) 

i-1 

Let  N  be  a  Poisson  random  variable  with  mean  value  n,  which  is  indep¬ 


endent  of  (X,  ,Y  ),(X_.Y.,),... 


.  In  the  sequel,  we  use  S  for  •  Notice 


That  B*s  are  disjoint,  we  see  that  for  t„f.( 

B 


^<“"1  ‘b  I 

i-1 

<*>  £  _ 

-  /  !^[E(exp([ 

£-0 

r '  tpY  ^ 

=  exp{n^  E[Ig(Xj)(e  “  -1)]}, 

N 

So  that,  {i  Ib(X.)Y.  -  njg  Q(x)dF|  ,B€*'}  is  a  group  of  mutually  independent 


variables .  Set 


Z(B,N)  =  *  "|b 


for  t  >  0,  notice  that  e^  -  t  >  e'^  +  t,  we  have 

P{|^  (Z(B,Nj|>  ine^}  5.  exp(-itne2)Eexp(t^  |Z(B,N)|) 

»  exp(-itnE3)ngg^^E{exp(t|Z(B.N)| )} 

<  exp(-itne3)ng^^-[Eexp(tZ(B,N)  +  Eexp(-tZ(B,N) ) ] 

ty  (18) 

=  exp(-itne3)ng^^.{exp[nEIg(X^)(e  ^-tY^-1)] 

+  exp[nEIg(Xj^)(e  +tYj^-l))} 

<  exp(-ttne3)ng^^-{2exp[nEIg(X^)(e  -t|Y^|-lJ} 

Cgh"^  T' 

<  exp(-itnE3)2  exp{nj  EtIg(X^)U^-tM-l )  ] } 

1  exp(-itnE3)e°^”^exp{n(e^-tM-l)}. 

Take  tC(0,l/M),  we  see  that  e^-tM-1  <  tV,  Hence,  we  can  take  t  >  0  such  that 
N 

?{S  I  y  V^i^^i  ■  "Ib  >  i^E.}  <  e 


(19) 


Write  A  ■  . . . ).  By  Jensen's  inequality,  for  t  >  0  we  have 


N 

•P{^  I  y  Ig(X^)Q(X.)  -  n|gQ(x)dF|  >' int^} 
i=l 

-  P{y  |E(Z(B,N)U)|  > 

<  P{E(y  iZ(B.N)( |A)  >  intg} 

<  exp(-itnCg)E{exp(tE(y  lZ(B,N) | | A] )} 

<  exp(-itne2)E{E[exp(ty  |Z(B,N) j  ) | A] } 


exp(-itnc^)E[exp(ty  jZ(B,N)|)] 


By  (.18)  and  (19),  we  can  take  t  >  0  such  that 
N 

p{y  )  y  Ig(X^)Q(X^)  -  n|gQ(x)dF|  >  inSg}  <  e 
i«l 


-Cgn 


(20) 


Note  that 


N  n 

/j  I 

i-1  i*l 

N  n 

y'l  y  ig(x.)Q(x.)  -  y  ig(x.)Q(x.)i  imin-ui 


i*l 


i»l 


From  (19)  -  (22),  (16)  and  (17)  follows,  and  (13)  is  proved.  It  remains 
to  prove  (14).  To  this  end,  we  need  only  to  prove 

V 

P{  j  Z(X^)>nE,}  <  e  ,  (23 

i=l 


where 


Z(u)  ”  h  I  (u)lQ(u)  -  Q(x)|dx 

pr  x+hA 


<  2Mh 


)dx  =  2MX(A)  =  C. 


Hence,  we  have 


N 

I  «X,) 

i-1 


;  Z(X.)|  <  Cg|N-n|, 
i*l 


N  n 

P{  ^  Z(X^)  -  ^  Z(X^)|  >  ^nc^} 

i»l  i=l 


<  P(|N-n(>nE2/(2Cg)}  <  e 


For  t  >  0,  we  have 


P{  ^  Z(X^)>inE2}  1  exp(-itnE2)E{exp(t  Z(X^)} 


exp{-  intE^  +  nJ(e^^^'*^-l)F(du)> . 


Take  tfe(0,l/Cg).  By  0<tZ(u)5_l  we  get 


tZ(u) 


l)F(du) 


£  2nt|z(u 


)F(du) 


Take  p  >  0  so  large  that  AcS^.  .  Then,  by  Lemma  4,  we  have 

U  9  P 


Z(u)  ■  h  ■  in(u)|dx 

-  ^^^0  nMS  iin(x)  -  m(u)|dx/X(S„  ,„) 


-►  0  as  h  -►  0,  for  almost  all  x(X). 


In  view  of  (24),  from  the  dominated  convergence  theorem,  we  see  that 


iim^^JZ(u)F(du)  =  0. 

By  (26)  -  (28),  we  can  take  t  >  0  sufficiently  small  such  that 
N 

P{  y  Z(X^)  >  ine^}  <  cxp(-Jnte2  +  o(nt)) 
i*»l 


From  (25)  and  (29),  we  obtain  (23),  and  (14)  follows.  Up  to  now,  the 
theorem  is  proved. 
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